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1.0  EXECUTIVE  SUMMARY 


Due  to  the  large  numbers  (up  to  tens  of  thousands)  of  possible  targets  identified  in  nominal 
unexploded  ordnanee  (UXO)  surveys,  effieient  and  reliable  maehine-aided  target  piekers  should 
be  used  to  identify  targets  for  subsequent  eharaeterization.  When  seleeting  anomalies,  the  goal  is 
to  identify  all  anomalous  features  that  may  be  eaused  by  UXO,  while  minimizing  operator  time 
and  eliminating  operator  bias.  To  faeilitate  advaneed  physies-based  modeling,  however,  the 
target  piekers  should  also  be  able  to  seleet  data  appropriate  to  the  target,  i.e.,  to  outline  or 
estimate  the  anomaly’s  spatial  extent.  The  eurrent  approaeh  to  target  seleetion  is  either  manual 
identifieation  or  amplitude  thresholding.  The  former  is  time-intensive,  not  elearly  defined,  and 
prone  to  operator  bias.  The  latter  is  sensitive  to  noise  and  is  prone  to  over-  or  under-pieking 
unless  judieious  oversight  is  exereised.  Neither  approaeh  provides  measures  for  estimating  the 
footprint  of  the  anomaly.  The  impaet  to  the  Department  of  Defense  (DoD)  is  obvious. 
Systematie,  fast,  and  robust  target  piekers  ean  save  money  and  produee  a  defensible  target  list 
eompared  to  the  eurrent  methods. 

This  projeet  evaluated  four  automatie  target  piekers  as  well  as  the  manual  method  and 
transitioned  them  to  the  user  eommunity  via  Oasis  montaj™  by  building  eustom  Geosoft 
Exeeutables  (GX).  Oasis  montaj™  is  a  geophysieal  data  proeessing  and  visualization  paekage 
developed  and  marketed  by  Geosoft  Ineorporated.  The  four  automatie  target  piekers  were:  (1)  a 
wavelet-based  deteetion  algorithm,  (2)  elustering  positive  and  negative  peaks,  (3)  a  dipole-based 
matehed  filter  (MF),  and  (4)  analytie  signal  (AS). 

The  demonstration  was  broken  up  into  two  phases.  The  first  phase  used  a  60-dipole  synthetie 
dataset  to  explore  the  parameter  spaee  and  optimize  the  algorithms  for  the  four  automatie  target 
piekers.  The  result  of  Phase  1  was  a  set  of  starting  parameters  that  was  used  in  Phase  2.  The 
seeond  phase  applied  the  target  piekers  to  seven  magnetie  datasets  using  the  parameters  output 
from  Phase  1  as  a  starting  point.  The  seven  datasets  possessed  different  signal  and  noise 
eharaeteristies  and  anomaly  densities.  Three  datasets  provided  from  the  U.S.  Army  Corps  of 
Engineers  (USAGE)— a  helieop ter- towed  wide  area  assessment  (WAA)  dataset,  a  vehiele-towed 
transeet  WAA  dataset,  and  vehiele-towed  Multisensor  Towed  Array  Deteetion  System 
(MTADS)  datasets  from  the  Aberdeen  Proving  Ground  (APG)  standardized  test  site  and  Target 
SI  at  Isleta  Pueblo  in  New  Mexieo— were  used  for  the  evaluation.  Beeause  eaeh  dataset  has  its 
own  unique  data  eharaeteristies,  the  starting  parameters  were  adjusted  iteratively  to  aehieve  the 
best  performanee.  The  knowledge  gained  from  Phase  1  was  used  to  guide  these  adjustments. 

The  primary  performanee  objeetives  were  to  deteet  >90%  of  the  ground  truthed  targets  for 
produetion  surveys  (>80%  for  WAA)  in  <%  the  time  needed  for  the  manual  method.  The 
deteetion  objeetive  was  met  by  some  of  the  automatie  piekers  on  some  of  the  datasets.  The  time 
objeetive  was  met  by  all  the  automatie  methods  for  all  the  datasets. 

A  stated  objeetive  in  the  DoD’s  Report  to  Congress  in  2001  was  to  develop  standards  and 
protoeols  for  navigation,  geoloeation,  data  aequisition  and  proeessing,  and  performanee  of  UXO 
teehnologies.  These  inelude  standard  software  and  visualization  tools  to  provide  regulatory  and 
publie  visibility  to  and  understanding  of  the  analysis  and  deeision  proeess  made  in  response 
aetivities  [1,2]. 
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The  detection  rates  for  the  automatic  methods  were  heavily  dependent  on  the  data  quality,  noise 
characteristics,  and  target  density.  In  general,  the  amount  of  geology  and  background  noise 
affected  the  performance  of  the  automatic  methods  the  most.  All  the  methods  require  a  threshold 
to  be  set  that  is  based  on  the  noise  characteristics  of  the  data.  Because  of  this,  automatic  methods 
will  pick  anomalies  with  signal  amplitudes  above  the  threshold  that  are  caused  by  noisy  data  or 
geology  and  not  pick  the  valid  anomalies  located  in  the  quieter  areas  that  are  below  the 
threshold.  Datasets  that  contained  a  consistent  background  noise  level  across  the  area,  as  seen  in 
the  Isleta  dataset,  performed  well,  with  all  methods  exceeding  the  90%  detection  rate. 
Conversely,  the  WAA  helicopter  dataset  contained  a  variety  of  noise  levels  due  to  geology, 
which  caused  50%  detection  rates  for  all  automatic  methods.  The  manual  method  has  an 
advantage  in  these  areas  because  the  analyst  can  set  a  lower  picking  threshold  and  dynamically 
filter  out  the  picks  that  are  obviously  due  to  geology.  This  ability  may  also  be  a  drawback 
because  it  may  introduce  operator  error  or  operator  bias.  The  other  main  drawback  to  the  manual 
method  is  the  time  required  to  pick  the  anomalies. 

Of  the  automatic  methods,  the  AS  and  wavelet  method  gave  the  best  results  overall,  but  each 
method  had  its  own  strengths  and  weaknesses,  and  the  best  method  was  very  data-dependent.  A 
general  observation  for  all  the  automatic  pickers  is  that  they  should  not  be  run  blindly.  The 
analyst  should  carefully  choose  their  parameters  and  analyze  the  results.  The  process  of 
iteratively  changing  parameters  and  visual  review  of  the  results  was  essential  in  selecting  the  best 
parameters. 

Implementation  of  the  automatic  picking  methods  used  in  this  demonstration  should 
considerably  reduce  the  time  and  thus  cost  required  to  pick  anomalies  when  compared  to  the 
manual  method.  The  amount  of  cost  savings  will  depend  on  the  data.  In  areas  with  isolated 
anomalies  and  low  background  noise  or  geology,  the  cost  savings  will  be  maximized  because  the 
automatic  pickers  are  able  to  detect  over  90%  of  the  anomalies  in  a  fraction  of  the  time  compared 
to  the  manual  method.  As  the  geologic  noise  increases  or  data  quality  decreases,  the  cost  savings 
will  diminish  because  the  missed  anomalies  will  need  to  be  filled  in  using  the  time-consuming 
manual  method,  but  cost  savings  still  should  be  significant. 

The  stakeholders  and  end  users  of  this  data  processing  and  analysis  technology  include  private 
contractors  who  conduct  geophysical  investigations  in  support  of  UXO  cleanup  programs  and 
governmental  employees  who  provide  technical  oversight.  This  demonstration  showed  the  data 
products  associated  with  this  analysis  approach  and  the  inherent  transparency  of  the  target 
picking  process.  This  basic  information  will  help  to  improve  the  results  of  future  geophysical 
investigations  conducted  by  others.  The  market  for  this  type  of  guidance  document  includes  all 
practicing  geophysical  service  firms  currently  working  in  the  UXO  industry. 
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2.0  TECHNOLOGY  DESCRIPTION 


2,1  TECHNOLOGY  DEVELOPMENTS  AND  APPLICATION 

The  user  community  would  benefit  from  an  algorithm  that  accurately  identifies  and  locates 
magnetic  anomalies  using  quantifiable  decision  criteria  and  limited  user  interactions.  The 
underlying  aim  is  to  reduce  analysts’  bias  because  it  is  not  predictable,  consistent,  or  correctable. 
This  project  made  and  compared  anomaly  declarations  for  each  dataset  chosen  using  the 
following  five  approaches:  (1)  a  wavelet-based  detection  algorithm,  (2)  clustering  positive  and 
negative  peaks,  (3)  a  dipole-based  MF,  (4)  AS,  and  (5)  manual  selection.  Each  approach  is 
described  below.  We  also  evaluated  how  each  method  can  be  used  to  identify  the  anomaly’s 
spatial  extent. 

Wavelet-Based  Algorithm.  Billings  and  Herrmann  (2003)  developed  a  magnetic  target  picking 
method  called  the  Automated  Wavelet  Detection  (AWD)  algorithm  [3].  In  that  method, 
individual  peaks  in  the  magnetic  data  are  followed  across  multiple  scales,  with  the  decay  in  peak 
amplitude  related  to  the  depth  to  the  source.  Nearby  positive  and  negative  peaks  in  the  image  are 
only  joined  together  if  they  satisfy  two  conditions:  (1)  the  peaks  must  have  comparable  depth 
estimates  and  (2)  the  peaks  must  move  towards  each  other  as  the  wavelet  scale  becomes  finer.  In 
this  way,  one  can  avoid  incorrectly  joining  the  peaks  from  nearby  dipoles.  In  the  last  stage  of  the 
algorithm,  the  amplitudes  of  the  peaks  and  their  relative  position  are  used  to  provide  an  initial 
estimate  of  the  dipole  parameters,  including  a  good  initial  estimate  of  the  size  of  the  region  to 
invert  about  each  anomaly. 

Rule-Based  Clusters.  Rule-based  clustering  approaches  threshold  the  magnetic  data  to  identify 
positive  and  negative  groups  and  then  associate  positive  and  negative  clusters  using  codified 
rules  to  form  anomalies.  The  degree  to  which  they  work  depends  on  the  noise  and  signal 
characteristics  of  the  data  and  on  details  of  the  rules  that  associate  negative  and  positive  clusters. 
We  have  identified  two  rule-based  schemes  that  have  been  developed  and  codified  under 
previous  research  efforts. 

The  first  is  an  automatic  target  picker  developed  during  Strategic  Environmental  Research  and 
Development  Program  (SERDP)  CU-1092  by  Blackhawk  Geometries,  Inc.  [4].  In  this  version, 
thresholds  for  positive  and  negative  data  are  set  by  the  user.  The  negative  and  positive  clusters 
are  then  identified,  and  each  negative  anomaly  is  associated  with  the  most  plausible  nearby 
positive  anomaly.  The  most  plausible  positive  anomaly  is  determined  by  a  formula  that  reflects 
the  distance,  magnitude  of  both  anomalies,  and  degree  of  consistency  with  the  local  magnetic 
declination.  Isolated  negative  anomalies  are  discarded  as  noise.  As  detailed  in  the  CU-1092 
project  report,  there  are  issues  with  regard  to  matching  positive  and  negative  lobes  in  cases 
where  the  anomalies  are  tightly  packed  and/or  overlapping.  There  are  also  issues  with  regard  to 
the  picker  dividing  one  anomaly  into  two  if  there  are  minor  amplitude  sags  (due  perhaps  to  inter¬ 
sensor  difference,  heading  errors,  or  gridding). 

The  second  rule-based  scheme  was  developed  and  codified  by  AETC  Incorporated  under  funding 
from  Naval  Explosive  Ordnance  Disposal  Technology  Division  (NAVEODTECHDIV)  in  1995. 
Similar  to  the  code  developed  under  CU-1092,  it  asks  the  user  to  identify  thresholds,  uses  the 
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thresholds  to  create  positive  and  negative  clusters,  and  then  associates  the  clusters  in  favor  of 
combinations  with  the  least  distance  between  centers. 

Dipole-Based  MF.  AETC  developed,  under  Environmental  Security  Technology  Certification 
Program  (ESTCP)  Project  199918,  a  Matched  Eilter  AutoProcessor  (MEAP)  for  magnetometer 
data  [5,  6].  This  algorithm  implements  a  ME  based  on  a  magnetic  dipole  signal.  After  the 
magnetometer  data  is  interpolated  onto  a  regular  x-y  grid,  it  is  automatically  convolved  with  a 
magnetic  dipole  signal  model  for  a  target  accompanied  by  a  search  over  the  unknown  parameters 
in  the  signal  model  (target  orientation,  moment,  etc.)  to  maximize  the  filter  output.  The  output  of 
the  ME  routine  is  a  surface  (grid)  that  peaks  directly  over  the  target  location.  Because  of  this,  it 
can  be  thresholded  to  identify  targets.  The  MEAP  approach  was  developed  in  the  Interactive 
Data  Eanguage  (IDE)  (Research  Systems,  Inc.)  but  has  subsequently  been  integrated  with  OASIS 
montaj™. 

Once  the  ME  output  is  calculated,  the  user  must  identify  individual  anomalies  by  picking  peaks 
above  the  background  noise.  The  spatial  extent  of  each  anomaly  can  be  estimated  by  using  the 
initial  fitted  parameters  output  during  the  match  fitting  process  to  calculate  a  boundary  that 
contains  a  certain  percentage  of  the  total  energy. 

Analytic  Signal.  The  AS  is  a  positive  quantity  derived  from  magnetometer  data.  It  can  easily  be 
thresholded  for  target  detection  and  is  commonly  used  by  commercial  practitioners.  The  AS  is 
the  square  root  of  the  sum  of  the  gradients  in  three  directions: 


AS  = 


where  the  z-gradient  is  formed  from  the  data  in  the  (x,y)  plane  by  upward  continuation  in  the 
Eourier  domain: 


dBjk) 

dz 


=  kB{k), 


k  =  271: !  X  . 


Once  the  AS  is  calculated,  the  user  must  identify  individual  anomalies  by  picking  peaks  above 
the  background  noise.  The  spatial  extent  of  each  anomaly  can  be  estimated  by  examining  the 
nature  of  the  anomaly  in  terms  of  its  inflection  points  or  change  in  total  energy. 

Manual  Anomaly  Selection.  As  a  baseline  measure,  we  manually  identified  individual  anomalies 
and  estimated  the  spatial  footprint  of  each  anomaly  using  an  experienced  analyst  from  NAEVA 
Geophysics.  The  analyst  was  asked  to  select  all  anomalies  above  the  noise  level  of  the  given 
dataset.  The  analyst  decided  on  the  noise  threshold  and  properly  documented  the  rationale 
behind  the  threshold.  The  analyst  was  free  to  use  the  preferred  analysis  environment. 
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2.2  PROCESS  DESCRIPTION 


This  demonstration  used  target  picking  algorithms  that  have  been  prototyped  and  tested  during 
previous  research  programs.  Three  of  the  algorithms  (AWD,  MF,  and  clustering)  have  been 
seamlessly  integrated  into  Oasis  montaj™  under  this  project.  The  AS  algorithm  is  included  with 
appropriately  licensed  versions  of  Oasis  montaj All  processing  and  analysis  performed  in  this 
demonstration  was  done  in  the  Oasis  montaj  environment. 

The  demonstration  was  broken  up  into  two  phases.  The  first  phase  used  a  60-dipole  synthetic 
dataset  (Figure  1)  to  explore  the  parameter  space  and  optimize  the  algorithms  for  the  four 
automatic  target  pickers.  The  synthetic  dataset  consisted  of  six  target  sizes  (20  mm,  40  mm, 
60  mm,  81  mm,  105  mm,  and  155  mm)  randomly  placed  at  depths  up  to  11  times  the  target 
diameter.  The  result  of  Phase  1  was  a  set  of  starting  parameters  that  was  used  in  Phase  2. 
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Figure  1.  Synthetic  Magnetic  Dataset  with  60  Targets. 

Phase  2  applied  the  target  pickers  to  the  seven  magnetic  datasets  used  in  this  demonstration  using 
the  parameters  output  from  Phase  1 .  Because  each  dataset  has  its  own  unique  data  characteristics, 
the  starting  parameters  were  adjusted  iteratively  using  knowledge  gained  from  Phase  1  to 
achieve  the  best  performance.  The  process  involved  running  the  algorithms  on  the  datasets  (for 
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large  datasets  a  small  portion  of  the  data  was  used)  and  ehanging  the  parameters  based  on  a 
visual  review  of  the  results.  Several  iterations  were  run  until  the  optimal  parameters  were 
ehosen.  The  visual  review  involved  looking  at  the  types  of  anomalies  that  were  visible  in  the 
data  and  changing  the  parameters  to  select  the  desired  anomalies.  We  also  looked  at  all  the 
automatic  picks  and  determined  if  they  were  valid  or  in  the  noise  levels  of  the  dataset  and 
adjusted  the  threshold  accordingly. 

2.3  PREVIOUS  TESTING  OF  THE  TECHNOLOGY 

The  AWD  algorithm  has  been  successfully  applied  at  the  University  of  British  Columbia  as  part 
of  the  work  funded  by  Engineer  Research  and  Development  Center  (ERDC)  (DAAD 19-00-0- 
0120).  The  algorithm  was  demonstrated  on  three  datasets  with  different  characteristics  [3].  In 
each  case,  the  method  rapidly  located  the  majority  of  dipole  anomalies  and  produced  accurate 
estimates  of  the  dipole  parameters. 

The  two  rule-based  clustering  algorithms  used  as  a  basis  for  the  clustering  algorithm  developed 
for  this  project  were  previously  tested  during  their  development  as  part  of  the  work  funded  by 
SERDP  (CU-1092)  and  NAVEODTECHDIV.  The  AETC  clustering  algorithm  was  also  tested 
on  data  collected  at  the  Badlands  Bombing  Range  (BBR)  on  the  Pine  Ridge  Reservation  in  South 
Dakota  and  at  Twentynine  Palms,  California  [7].  The  results  of  this  test  showed  the  automatic 
processor  detecting  about  90-100%  of  the  objects  present  with  the  most  significant  detection 
factor  being  target  density.  In  low  density  regions,  almost  100%  of  the  objects  were  found, 
whereas  in  high  density  areas  the  detection  rate  fell  to  90%.  It  was  estimated  that  the  automatic 
processor  could  analyze  about  four  times  more  data  than  the  manual  processor  in  the  same 
amount  of  time. 

Previous  testing  of  the  ME  algorithm  included  MTADS  magnetometer  survey  data  over  several 
sites  (Twentynine  Palms,  Blossom  Point,  and  Buckley  Eield).  The  algorithm  was  also  tested  on 
the  three  1-hectacre  areas  of  the  2000  Advanced  UXO  Detection/Discrimination  Technology 
Demonstration  at  Jefferson  Proving  Ground  (JPG).  The  ME  did  a  good  job  at  isolating  potential 
targets  at  JPG  [5].  The  filter  output  peaked  over  the  targets,  which  helped  reduce  the  ambiguity 
of  the  exact  target  location.  The  low  probability  of  detection  (Pd)  (80%)  was  caused  by  data 
gaps  and  the  difficulty  in  finding  20  mm  using  magnetometer  data  with  a  sensor  spacing  of 
25  cm. 

The  AS  is  included  with  appropriately  licensed  versions  of  Oasis  montaj^*^  and  is  commonly 
used  by  commercial  analysts.  As  such,  it  has  undergone  internal  testing  of  the  algorithm  at 
Geosoft  and  extensive  practical  testing  by  the  user  community. 

2.4  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 

This  demonstration  used  already  collected  survey  data  to  test  four  automatic  target  pickers  and 
transitioned  the  algorithms  to  the  user  community  through  Oasis  montaj™.  The  main  advantage 
of  using  automatic  methods  to  pick  anomalies  is  the  time  it  can  save.  On  large  remediation 
projects  with  thousands  of  anomalies,  the  manual  method  is  very  slow.  Another  aspect  of  the 
manual  method  is  its  subjective  nature.  Different  data  analysts  may  have  different  criteria  on 
what  is  and  is  not  an  anomaly.  It  may  be  based  on  signal  level  and  signal  shape,  which  is 
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directly  affected  by  the  way  the  data  is  displayed.  The  target  density  of  the  data  may  also  affect 
the  analyst’s  decision.  In  areas  of  low  density,  the  analyst  may  be  prone  to  pick  smaller  and 
weaker  anomalies  for  the  sake  of  picking  something.  Conversely,  in  high-density  areas  the 
analyst  may  set  his  threshold  higher.  Based  on  these  two  difficulties,  automatic  target  pickers 
are  desirable  because  they  would  be  limited  only  by  the  speed  of  the  computer  and  can  be  set  to 
select  targets  using  fixed  criteria. 

Previous  testing  of  the  automatic  pickers  has  shown  limitations  to  the  technology.  The  clustering 
and  AS  algorithms  may  have  trouble  with  low  signal-to-noise  signatures  and  high  background 
noise.  In  these  areas,  a  low  threshold  must  be  chosen  to  pick  the  weaker  anomalies,  but  by 
lowering  the  threshold,  the  automatic  pickers  will  pick  many  anomalies  that  are  caused  by  noise 
in  the  data. 

The  MF  algorithm  cannot  filter  a  point  if  there  are  any  missing  data  in  the  filter  box.  Thus,  the 
edges  of  a  survey  site  cannot  be  filtered  as  well  as  any  interior  regions  that  are  missing  data. 
Therefore,  a  small  data  gap  of  0.5x5  m  becomes  an  unfilterable  region  of  3.5x8  m  if  a  3  m  filter 
box  is  used.  For  this  reason  the  match  filter  algorithm  cannot  be  used  on  the  WAA  vehicular 
transect  data  because  the  width  of  the  transect  is  only  1.75  m. 

The  AS  and  AWD  algorithms  map  and  interpolate  the  magnetic  data  to  a  regular  x-y  grid  and 
apply  their  respective  filters  to  the  grid.  For  normal  surveys,  this  would  not  be  an  issue  but 
applying  them  to  transect  data  may  require  some  optimization.  Because  transects  can  cover  large 
areas  in  a  single  sortie,  the  resultant  grid  can  get  very  large  given  that  the  grid  produced  is  based 
on  the  minimum  and  maximum  XY  values.  If  the  transect  lines  are  relatively  straight  along  one 
geographic  heading,  the  data  can  be  rotated  prior  to  creating  the  grid  such  that  the  lines  are 
oriented  along  either  the  grid  rows  or  columns. 

The  AS  and  MF  algorithms  output  an  anomalous  surface  and  not  the  locations  of  the  anomalies. 
Therefore,  a  second  program  must  be  run  to  detect  the  peaks  in  the  surfaces.  Thus,  the  success 
of  these  methods  is  partially  dependent  on  the  peak  picking  program. 

The  clustering  and  AWD  algorithms  have  several  parameters  that  need  to  be  set.  The  option  to 
change  many  of  the  parameters  allows  the  user  the  flexibility  to  tune  the  algorithms  to  the 
individual  data  characteristics.  However,  this  flexibility  adds  to  the  complexity  and  setup  time  to 
ensure  that  the  proper  parameters  are  picked  to  achieve  optimum  performance  of  the  algorithms. 
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DEMONSTRATION  DESIGN 


3.1  PERFORMANCE  OBJECTIVES 

Table  1  and  Table  2  outline  the  performance  objectives  for  the  production  and  WAA  surveys, 
respectively.  The  values  of  the  quantitative  performance  objectives  for  the  production  surveys 
were  chosen  to  measure  how  practical  the  methods  are  for  use  in  the  “real”  world.  The  picks  by 
the  different  methods  were  compared  to  the  ground  truth  for  the  production  surveys  by 
calculating  the  Pd  defined  as  (number  of  detections  picked  by  an  automatic  picker  that  are 
coincident  to  a  ground  truth  object  /  number  of  ground  truthed  objects).  The  background  alarm 
rate  (rBAR)  defined  as  (number  of  detections  picked  by  automatic  picker  not  caused  by  ground 
truthed  object  /  arbitrary  number  (in  order  to  conceal  the  absolute  number  of  truth  items)  was 
calculated  only  for  the  APG  dataset.  This  was  because  none  of  the  other  datasets  contained 
complete  ground  truth  so  the  additional  picks  may  or  may  not  have  been  caused  by  a  valid 
object.  Because  the  calculation  used  an  arbitrary  number,  the  actual  performance  metric  is 
meaningless  but  useful  information  is  gained  by  comparing  the  number  among  the  different 
target  picking  methods.  The  manual  method  was  used  as  the  baseline  to  judge  the  efficiency  of 
the  different  methods.  Our  goal  is  for  the  processing  time  (time  required  to  run  automatic  picker 
/  time  required  to  manually  pick  the  anomalies)  to  be  less  than  0.25  hr.  This  would  allow  plenty 
of  time  to  manually  alter  the  automatic  picks  to  achieve  the  best  possible  performance  and  still 
complete  the  analysis  in  less  time  than  in  using  only  the  manual  method. 

We  compared  the  fitted  parameters  using  results  from  the  automatic  methods  and  the  manual 
method  to  ground  truth  information  for  80  targets  at  APG.  The  XY  location  and  a  boundary  file 
that  estimates  the  anomaly’s  spatial  extent  were  used  as  inputs  to  advanced  physics-based 
inversion  routines  that  output  target  features  such  as  XY  location,  depth,  and  size.  The  detection 
location  accuracy  is  defined  as  the  horizontal  distance  between  the  picked  location  that  is  output 
directly  from  the  target  picking  algorithms  and  the  ground  truth  location.  The  characterization 
location  accuracy  is  the  horizontal  distance  between  the  XY  position  output  from  the  inversion 
routines  and  the  ground  truth  location.  Therefore,  assuming  good  data  quality  the 
characterization  location  should  be  more  accurate  than  the  detection  location.  The 
characterization  size  accuracy  is  defined  as  (the  absolute  difference  between  the  fitted  target 
diameter  and  the  ground  truth  diameter)/(ground  truth  diameter).  The  location,  depth,  and  size 
metrics  were  set  to  values  that  would  enable  the  dig  teams  to  easily  find  the  target. 

The  WAA  surveys  have  similar  performance  criteria  as  the  production  surveys,  but  because  of 
differing  objectives  the  expected  performance  will  be  different.  The  general  objective  of  the 
WAA  surveys  is  to  delineate  munitions  response  sites,  support  regulatory  disposition  of  non¬ 
munitions  response  sites  and  provide  reliable  data  to  support  risk  analysis  and  remedial  cost 
estimation.  The  Pd  metric  is  lower  because  of  the  different  objectives.  To  achieve  these  goals, 
airborne  magnetometer  data  and  vehicle -based  magnetic  transect  data  were  collected.  These 
datasets  have  much  lower  data  densities,  which  results  in  the  location  metrics  being  higher  than 
for  production  surveys.  Also,  ground  truth  data  was  not  available  for  the  WAA  data  sets  so  the 
Pd  and  probability  of  false  alarm  (Pfa)  were  calculated  using  the  manual  picks  as  the  baseline. 
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Table  1,  Performance  Objectives  for  Production  Surveys. 


Primary 

Performance 

Criteria 

Description 

Expected  Performance 
(metric) 

Actual 
Performance 
Objective  Met? 

Qualitative  Performance  Criteria 

Ease  of  use 

Processing  flow  and  the 
anticipated  skill  level  required 

Limited  training  required  for 
users  experienced  in  Oasis 
montaj™ 

Yes 

Robustness 

Analysts’  experience  related  to 
bugs  and  program  deficiencies 

Analysis  not  hindered  by  bugs  or 
deficiencies 

Yes 

Quantitative  Performance  Criteria 

Pd  (ground  truth) 

(number  of  detections  picked  by 
automatic  picker  that  are 
coincident  to  ground  truth 
target)/(number  of  ground  truth 
targets) 

>0.9 

Yes/No 

rBAR  (APG  only) 

(number  of  detections  picked  by 
automatic  picker  but  not 
matching  ground  truth  object)/ 
(arbitrary  number) 

NA 

NA 

Setup  time 

Time  required  to  setup  optimum 
parameters 

<4  hours 

Yes/No 

Proeessing  time 

(Time  required  to  run  automatic 
picker)/(time  for  commercial 
expert  to  pick  anomalies) 

<0.25  hr. 

Yes 

Detection  location 
accuracy 

The  horizontal  distance 
between  picked  and  ground 
truth  location 

90%  <0.5  m 

Yes/No 

Charaeterization  size 
aeeuraey  (APG  only) 

(The  absolute  difference 
between  the  fitted  target 
diameter  and  the  ground  truth 
diameter)/(ground  truth 
diameter) 

90%  <0.3  m 

No 

Charaeterization 
loeation  aeeuraey  (APG 
only) 

The  horizontal  distance 
between  the  fitted  XY  position 
and  ground  truth  location 

90%  <0.3  m 

Yes 

Charaeterization  depth 
aeeuraey  (APG  only) 

The  difference  between  the 
fitted  depth  and  ground  truth 
depth 

90  %  <0.3  m 

Yes 
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Table  2.  Performance  Objectives  for  WAA  Surveys. 


Primary 

Performance 

Criteria 

Description 

Expected  Performance 
(metric) 

Actual 
Performance 
Objective  Met? 

Qualitative  Performance  Criteria 

Ease  of  use 

Processing  flow  and  the  anticipated 
skill  level  required 

Limited  training  required 
for  users  experienced  in 

Oasis  montaj™ 

Yes 

Robustness 

Analysts’  experience  related  to  bugs 
and  program  deficiencies 

Analysis  not  hindered  by 
bugs  or  deficiencies 

Yes 

Quantitative  Performance  Criteria 

Pd  (manual  pieks) 

(number  of  detections  picked  by 
automatic  picker  that  are  coincident  to 
manual  pick)/(number  of  manual  picks) 

>0.8 

No 

Pfa 

(number  of  detections  picked  by 
automatic  picker  but  not  matching 
manual  pick)/(number  of  manual  picks) 

<0.25 

Yes/No 

Setup  time 

Time  required  to  setup  optimum 
parameters 

<4  hours 

Yes/No 

Proeessing  time 

(Time  required  to  run  automatic 
picker)/(time  for  commercial  expert  to 
pick  anomalies) 

<0.25  hr. 

Yes 

Detection  location 
accuracy 

The  horizontal  distance  between  picked 
and  manual  pick  location 

90%  <1.5  m 

Yes 

3.2  SELECTING  TEST  SITES 

This  demonstration  was  performed  in  the  Cary,  North  Carolina,  offiee  of  SAIC  (formerly  AETC) 
on  data  that  was  previously  eolleeted  on  government  ranges  for  other  purposes.  There  are  thus 
no  regulatory,  health,  or  safety  issues  affeeting  this  demonstration. 

The  USAGE  was  asked  to  provide  examples  of  datasets  that  are  representative  of  that  aequired 
by  their  eontraetors  and  that  support  our  requirements.  Our  eriteria  for  the  demonstration  data 
sets  were  the  following:  a)  a  range  of  data  density  and  target  sizes,  and  a  variety  of  target  types 
were  needed  to  explore  the  eapabilities  and  performanee  of  the  four  automatie  piekers;  b)  a  range 
of  signal-to-noise  ratio  (SNR)  from  marginal  to  exeellent  in  order  to  derive  eonclusions  that  ean 
be  extrapolated  for  future  use;  e)  data  was  eolleeted  from  different  platforms,  ineluding 
vehieular,  man  portable,  and  airborne;  d)  datasets  that  are  of  both  standard  eontractor  and 
researeh  quality  in  terms  of  sensor  noise,  lag,  ete.;  e)  data  sets  eontaining  at  least  100,  but  ideally 
no  more  than  500,  anomalies;  f)  survey  objeetive  (WAA  or  produetion);  and  g)  available  ground 
truth  information.  Complete  ground  truth  information  was  desirable  for  all  datasets  but  not 
absolutely  neeessary.  When  available,  we  foeused  our  analysis  on  comparing  the  anomaly  picks 
to  the  ground  truth  information.  If  this  information  was  not  available,  we  used  the  manual  picks 
as  the  baseline  for  comparison  to  the  automatic  picks.  Accurate  ground  truth  information 
(location,  burial  depth,  size,  and  description)  was  needed  for  50-100  targets  in  order  to  compare 
the  location  accuracy  and  the  extracted  spatial  footprint  of  the  anomaly  for  the  different  picking 
algorithms. 

Table  3  summarizes  how  well  the  datasets  fit  our  demonstration  criteria. 
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Table  3,  Criteria  and  Datasets, 


Dataset/ 

Criteria 

Naval 
Research 
Laboratory 
(NRL) 
MTADS 
APG  June 
2004 

USAGE 

JPG 

USAGE 
Geophysical 
Proveout 
(GPO)  Data 

WAA 

Pueblo 

BT4 

Ground 

Transects 

WAA 

Pueblo 

BT4 

Airborne 

Isleta 

Vehicle 

Seaside 

Target 

sizes 

20  mm 

40  mm 

57  mm 

60  mm 

81  mm 

2.75  in 

105  mm 

155  mm 

bdu28 

blu26 

M42 

MK118 

OE  scrap 

4.5  in 
head, 
mortar 
burster 

MKIIgrenade 
MKII17#bomb 
MKI  25#bomb 
range  clutter 

M3  8 

MKI  5 

GP  Bomb 

M3  8 

MK15 

GP  Bomb 

60  mm 

81  mm 

2.75  in 

105  mm 
BDU33 
MK76 

M3  8 

GP  bomb 

37  mm, 

60  mm, 

3.5  in 

M29, 

Grenade 

fuze. 

Non  OE 
scrap 

Target 

density 

low-high 

low-high 

low-med 

low-med 

low-med 

low-med 

low-high 

Platform 

vehicle 

handheld 

handheld 

vehicle 

airborne 

vehicle 

handheld 

Data 

quality 

high 

[research] 

low 

[standard] 

med 

[standard] 

high 

[research] 

high 

[research] 

high 

[research] 

med 

[standard] 

SNR 

low-high 

low-high 

low-high 

low-high 

low-high 

low-high 

low-high 

Geologic 

noise 

low-med 

low 

low-high 

low-med 

med-high 

low-med 

low-med 

Number 
of  ground 
truthed 
anomalies 

All,  but  not 
released  to 
public 

253 

33 

2 

0 

150 

481 

Survey 

objective 

production 

production 

production 

WAA 

WAA 

production 

production 

Picking 

methods 

AWD,  AS, 

MF, 

clustering, 

manual 

AWD,  AS, 
MF, 

clustering, 

manual 

AWD,  AS, 

MF,  clustering, 
manual 

AWD,  AS, 
clustering, 
manual 

AWD,  AS, 
MF, 

clustering, 

manual 

AWD,  AS, 
MF, 

clustering, 

manual 

AWD,  AS, 
MF, 

clustering, 

manual 

Three  of  the  datasets  were  aequired  using  the  vehicle-towed  MTADS  by  NRL  and  AETC. 
MTADS  hardware  consists  of  a  low  magnetic  signature  vehicle  that  is  used  to  tow  an  array  of 
eight  magnetic  sensors  that  are  spaced  .25  m  apart.  The  sensor  positions  were  determined  using 
real-time  kinematic  (RTK)  Global  Positioning  System  (GPS)  receivers.  The  three  data  sets  were 
collected  at  1)  open  field  grid  of  the  APG  standardized  test  site  in  June  2004;  2)  Target  SI  on  the 
Pueblo  of  Isleta  near  Albuquerque,  New  Mexico,  in  February  2003;  and  3)  area  near  Bombing 
Target  4  (BT-4)  at  Pueblo  PBR  in  Colorado  in  September  2005. 

Another  data  set  was  collected  by  Sky  Research  and  AETC  using  the  airborne  MTADS  system. 
The  system  hardware  includes  an  array  of  seven  magnetometers  spaced  1.5  m  apart  in  a  9  m 
boom  mounted  on  a  Bell  206E  helicopter.  The  sensor  positions  were  determined  using  RTK  GPS 
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technology.  The  data  collected  was  in  the  same  vicinity  as  the  above-mentioned  vehicle -towed 
MTADS  data  near  BT-4  at  Pueblo  PBR. 

USAGE  contractors  acquired  the  remaining  three  datasets.  American  Technologies  Incorporated 
acquired  data  on  a  removal  action  over  a  300-acre  area  at  JPG.  The  data  were  collected  using  the 
handheld  Geometric  G-858G  vertical  gradiometer  system  in  fiducial  positioning  mode.  Two 
magnetometers  were  vertically  separated  by  2  feet  and  data  were  collected  using  a  lane  spacing 
of  2.5  feet.  NAEVA  Geophysics  collected  magnetic  data  over  a  GPO.  (The  USAGE  has 
requested  that  the  site  specifics  be  confidential.)  The  data  were  collected  using  a  single  handheld 
magnetometer  with  a  line  spacing  of  2  feet  and  positioned  using  RTK  GPS.  The  last  dataset  was 
collected  by  Parsons  over  the  Seaside  area  at  the  Eormer  Port  Ord.  The  data  were  collected  using 
a  handheld  array  of  four  magnetometers  spread  cross  line  at  2-foot  intervals  and  positioned  using 
RTK  GPS. 

3.3  TEST  SITE  HISTORY/CHARACTERISTICS 

The  area  of  the  Aberdeen  Test  Site  is  adjacent  to  the  Trench  Warfare  facility  at  the  APG.  The 
APG  Standardized  Test  Site  is  located  within  a  secured  range  area  of  the  APG.  The  Aberdeen 
area  of  APG  is  located  approximately  30  miles  northeast  of  Baltimore  at  the  northern  end  of  the 
Ghesapeake  Bay.  The  Standardized  Test  Site  encompasses  17  acres  of  upland  and  lowland  fiats, 
woods,  and  wetlands.  The  test  site  is  divided  into  calibration  lanes,  blind  grid  test  grid,  open 
field,  mogul,  and  wooded  areas.  Additional  details  regarding  the  layout  of  the  APG 
Standardized  Test  Site  can  be  found  at  http;//aec. army.mil/usaec/technology/uxo03c01.html. 

The  total  area  of  the  former  Pueblo  Precision  Bombing  and  Pattern  Gunnery  Range  #2  is  67,769 
acres,  and  it  is  located  approximately  20  miles  south  of  Ea  Junta,  Golorado,  in  Otero  Gounty. 
The  closest  community  is  Ea  Junta,  a  rural  town  with  a  population  of  7,637.  The  Munitions 
Response  Area  (MRA)  was  used  by  local  populations  for  cattle  grazing  until  the  War 
Department  assumed  control  of  the  lands  to  construct  the  Pueblo  Precision  Bombing  and  Pattern 
Gunnery  Range  #2  (1942  to  1946). 

The  Pueblo  of  Isleta  is  located  in  north-central  New  Mexico,  approximately  10  miles  south  of 
Albuquerque.  The  reservation  is  bordered  on  the  north  by  the  Sandia  Military  Reservation, 
which  includes  Kirtland  Air  Eorce  Base,  the  Manzano  Mountains  on  the  east,  and  the  Rio  Puerco 
and  Eaguna  Pueblo  Reservation  on  the  west.  The  area  that  contains  target  SI  comprises 
approximately  7,000  acres  that  were  leased  from  the  tribe  in  the  1950s  for  use  as  a  target 
bombing  range  for  aircraft  from  Kirtland.  Documentation  in  Bureau  of  Indian  Affairs  files 
indicates  that  this  area  was  used  as  a  practice  bombing  range  from  1956  to  1961  to  determine  the 
performance  of  fast  aircraft  during  bombing  runs.  In  the  1960s,  Kirtland  collected  and  piled 
visible  ordnance  debris  for  removal.  Up  to  2  tons  of  practice  bombs  and  ordnance  waste  per  acre 
were  removed  but  no  explosive  ordnance  was  found. 

JPG,  a  55,265-acre  facility  was  established  in  December  1940,  fired  its  first  round  5  months 
later,  and  operated  until  1995.  JPG’s  primary  mission  was  to  perform  production  and  post¬ 
production  tests  of  conventional  ammunition  components  and  other  ordnance  items  and  to 
conduct  tests  of  propellant  ammunition/weapons  systems  and  components  for  the  U.S.  Army. 
JPG  is  located  in  southeastern  Indiana,  approximately  8  miles  north  of  the  Indiana-Kentucky 
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border  and  about  5  miles  north  of  Madison,  Indiana.  Lands  surrounding  JPG  are  predominantly 
farmland  and  woodlands,  with  some  small  towns  and  rural  residential  land  use  nearby. 


The  former  Fort  Ord  is  loeated  80  miles  south  of  San  Franeiseo  and  oecupies  approximately 
28,000  aeres  adjaeent  to  Monterey  Bay  and  the  eities  of  Marina,  Seaside,  Sand  City,  Del  Rey 
Oaks,  and  Monterey.  Fort  Ord  beeame  a  training  installation  in  1917  and  was  used  to  train  Army 
infantry,  eavalry,  and  field  artillery  divisions  for  WWI,  WWII,  the  Korean  War,  the  Vietnam 
War,  and  Desert  Storm. 

3.4  PHYSICAL  SETUP  AND  OPERATION 

This  data  analysis  demonstration  utilized  data  previously  eolleeted  on  government  ranges  for 
other  purposes.  As  sueh,  no  site  preparation  or  installations  were  required  for  this 
demonstration.  Key  ehronologieal  dates  with  regard  to  the  overall  program  are  listed  in  Table  4. 

Table  4,  Target  Picking  Demonstration  Schedule, 


Date 

Action 

November  2006 

Reeeive  approval  for  Demonstration  Plan 

Deeember  1-15,  2006 

Phase  1  testing  using  a  60-dipole  synthetie  dataset 

Deeember  15,  2006  -  January  15,  2007 

Applied  AS  and  MF  algorithms  to  all  demonstration  datasets 

January  15  -  Febmary  1,  2007 

Applied  elustering  algorithm  to  3  of  7  demonstration  datasets 

January  15  -  Febmary  15,  2007 

Applied  AWD  algorithm  to  all  demonstration  datasets 

May  2007 

Presented  preliminary  results  at  In-Progress  Review  (IPR) 

July  2007 

Reeeived  new  Teehnieal  Assistanee  Agreement  (TAA)  with  Geosoft, 
fixed  memory -related  bug  in  elustering  algorithm,  and  analyzed  four 
remaining  datasets 

Mareh  2008 

Reeeived  performanee  results  from  IDA  for  APG  dataset 

Mareh  2008 

Submitted  ESTCP  Draft  Final  Report 
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4.0  PERFORMANCE  ASSESSMENT 


4,1  PERFORMANCE  DATA 

The  results  of  the  first  phase  of  the  demonstration  using  the  60-dipole  synthetie  dataset  are 
summarized  in  Figure  2,  which  shows  that  none  of  the  automatic  target  pickers  detected  all  the 
targets.  The  AS  and  the  wavelet  produced  the  best  results  followed  by  the  clustering  algorithm 
and  finally  the  MF. 

Synthetic  60  Dipole  ROC 


#  false  alarms 

Figure  2,  Receiver  Operating  Characteristic  (ROC)  Curve  of  the  Different  Target  Pickers 
for  the  Synthetic  60-Dipole  Data,  (The  symbols  along  each  line  represent  the  analyst’s 

threshold  value.) 

Detection  and  location  accuracy  and  target  picking  time  for  the  live  site  datasets  analyzed  in 
Phase  2  are  reported  in  Table  5  to  Table  7  and  Figure  3  to  Figure  6.  All  the  production  survey 
datasets  compared  the  declarations  made  by  the  different  methods  to  the  available  ground  truth  to 
calculate  the  Pd.  All  the  WAA  survey  data  sets  compared  the  declarations  made  by  the 
automatic  pickers  to  the  picks  made  by  the  commercial  expert  to  calculate  the  Pd  and  Pfa 
because  ground  truth  did  not  exist.  The  Pds  for  the  production  survey  datasets  ranged  from  .15 
to  .99.  The  variety  can  be  mainly  attributed  to  the  differences  in  data  and  available  ground  truth. 
The  Isleta  demonstration  showed  very  good  results  with  Pds  for  all  methods  above  .90,  whereas 
the  Seaside  demonstration  had  low  Pds  because  the  ground  truth  contained  many  objects  with  a 
weight  less  than  a  20  mm  projectile.  Over  90%  of  the  missed  targets  at  Seaside  were  attributed 
to  these  small  items.  As  part  of  our  analysis,  we  plotted  the  ground  truth  sorted  by  decreasing 
AS  amplitude  versus  the  number  of  targets  detected  by  each  picking  method  (Figure  3  to 
Figure  6).  This  would  ideally  give  a  straight  line  along  the  diagonal  if  all  the  ground  truth  were 
detected.  By  sorting  according  to  signal  strength,  we  are  able  to  visually  see  if  there  are  trends  in 
the  types  of  targets  missed  for  the  different  datasets.  It  also  gives  a  visual  sanity  check  of  the 
thresholds  selected  for  the  different  methods. 
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The  actual  time  it  took  to  set  up  the  optimum  parameters  and  run  the  automatic  target  pickers 
was  recorded  for  each  algorithm  and  dataset.  We  also  measured  the  setup  time  and  processing 
time  separately  because  the  setup  time  will  be  major  component  of  the  total  time  for  the 
relatively  small  datasets  we  used  in  this  demonstration. 

Table  5,  Detection  and  Location  Accuracy— USAGE  GPO,  Isleta,  and  Seaside, 


USAGE  GPO 

Isleta 

Seaside 

Parameter  / 

% 

% 

% 

Total  # 

within 

Total  # 

within 

Total  # 

within 

Method 

picks 

Pd 

,5  m 

picks 

Pd 

,5  m 

picks 

Pd 

,5  m 

Ground  truth 

28 

145 

412 

Manual 

36 

0.89 

88 

1,466 

0.98 

80 

205 

0.33 

84 

Clustering 

27 

0.57 

63 

758 

0.98 

84 

160 

0.25 

69 

Wavelet 

39 

0.75 

81 

1,454 

0.99 

86 

616 

0.60 

87 

Matched  filter 

14 

0.36 

70 

472 

0.90 

86 

123 

0.15 

85 

Analytic  signal 

50 

0.79 

95 

1,278 

0.98 

82 

346 

0.53 

83 

Table  3,  Detection  and  Location  Accuracy— JPG  and  APG. 


Parameter  / 
Method 

JPG 

APG 

Total  # 
picks 

Pd 

%  within 
,5  m 

Total  # 
picks 

Pd 

rBAR 

Average  miss 
Distance  (m) 

Ground  truth 

190 

NA 

Manual 

532 

0.95 

84 

707 

0.9 

0.58 

.204 

Clustering 

359 

0.92 

86 

444 

0.8 

0.335 

.196 

Wavelet 

491 

0.95 

89 

743 

0.85 

0.615 

.212 

Matched  filter 

309 

0.72 

87 

206 

0.7 

0.105 

.171 

Analytic  signal 

393 

0.92 

82 

620 

0.9 

0.495 

.225 

Table  4.  Detection  and  Location  Accuracy— WAA  Surveys, 


WAA  Pueblo  Airborne 

WAA  Pueblo  Transects 

Parameter  / 
Method 

Total  # 
picks 

Pd 

Pfa 

%  within 
1,5  m 

Total  # 
picks 

Pd 

Pfa 

%  within 
,5  m 

Manual 

383 

887 

Clustering 

387 

0.55 

0.46 

92 

454 

0.49 

0.02 

95 

Wavelet 

547 

0.53 

0.90 

92 

1,049 

0.76 

0.42 

92 

Matched  filter 

323 

0.50 

0.34 

94 

NA 

NA 

NA 

NA 

Analytic  signal 

397 

0.53 

0.50 

99 

505 

0.55 

0.02 

100 
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USAGE  GPO  -  #  ground  truthed  objects  detected 


Targets  sorted  by  decreasing  amplitude 


'-*-AWD 

-■—Analytic  signal 
Clustering 
Matched  filter 
'-^Manual 


Seaside  -  #  ground  truthed  objects  detected 


Targets  sorted  by  decreasing  amplitude 


JPG  -  #  ground  truthed  objects  detected 


-AWD 

-Analytic  signal 
Clustering 
Matched  fitter 
-Manual 


Targets  sorted  by  decreasing  amplitude 


Figure  3,  Graphs  Showing  the  Number  of  Ground  Truth  Objects  Detected  by  Each 
Method  as  a  Function  of  Signal  Amplitude  (USAGE  GPO  [top  left],  Isleta  [top  right],  Seaside 

[bottom  left],  and  JPG  [bottom  right]). 


Figure  4,  Graphs  Showing  Pd  Versus  the  Number  of  False  Alarms  (automatic  picks  not 
matching  manual  picks)  (Pueblo  Airborne  [left]  and  Pueblo  Transects  [right]). 
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APG  ROC  (Response) 


rBAR 


- AWD 

—  Analytic  signal 
Clustering 
Matched  filter 

—  Manual 


Figure  5,  Receiver  Operating  Curve  Showing  the  Number  of  Ground  Truth  Objects 
Detected  by  Each  Method  Versus  Background  Alarm  Rate  (APG), 


Target  picking  total  time 


Target  picking  processing  time 
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□  Manual 

■  Clustering 

□  Wavelet 

□  Matched  filter 

■  Analytic  signal 


Figure  6,  Graph  Showing  the  Total  Time  (includes  setup  time)  and  Processing  Time  to 
Analyze  the  Data  Sets  for  Each  Target  Picking  Method, 

4,2  PERFORMANCE  CRITERIA 

This  demonstration  resulted  in  32  sets  of  scores.  A  set  of  scores  for  eaeh  target  picking  method 
(five)  run  on  each  of  the  production  survey  datasets  gave  25  scores.  The  WAA  airborne  data  was 
compared  to  the  manual  picks  and  produced  four  sets  of  scores.  Finally,  the  WAA  transect  data 
consisted  of  only  three  sets  of  scores  because  the  MF  algorithm  was  not  applied  and  the  other 
automatic  methods  were  compared  to  the  manual  picks.  Performance  results  for  the  different 
datasets  are  summarized  in  Table  8.  All  the  production  survey  datasets  compared  the 
declarations  made  by  the  different  methods  to  the  available  ground  truth  to  calculate  the  Pd  [8]. 
All  the  WAA  survey  datasets  compared  the  declarations  made  by  the  automatic  pickers  to  the 
pieks  made  by  the  commereial  expert  to  ealeulate  the  Pd  and  Pfa  because  ground  truth  did  not 
exist.  The  maximum  Pd  is  1,  but  the  maximum  Pfa  eould  be  greater  than  1  beeause  it  is  based  on 
the  number  of  manual  picks.  The  ground  truth  for  APG  is  well  documented.  For  this  reason  the 
rBAR  and  characterization  target  parameter  accuracy  (location,  depth,  and  size)  were  also 
calculated  for  the  APG  dataset.  Because  only  a  select  number  of  targets  has  been  released  to  the 
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public,  the  scoring  for  the  APG  dataset  was  done  by  analysts  from  the  Institute  for  Defense 
Analyses. 


Table  8.  Performance  Criteria  for  This  Demonstration, 


Performance 

Criterion 

Expected 

Performance 

(metric) 

Performance  Confirmation 
Method 

Actual  Performance 

Ease  of  use 

Limited  training 
required  for  users 
experieneed  in  Oasis 
montaj™ 

General  observations 

Some  training  required  to 
set  proper  parameters  for 
data  proeessor  experieneed 
in  Oasis  montaj™ 

Robustness 

Analysis  not  hindered 
by  bugs  or  defieieneies. 

General  observations 

Analysis  flow  not  seriously 
interrupted  by  bugs 

Pd-produetion  surveys 
(ground  truth) 

>0.9 

Comparison  to  ground  truth  data 

0.15-0.99 

Pd-WAA  surveys 
(manual  pieks) 

>0.8 

Comparison  to  manual  pieks 

0.490-0.76 

rBAR  (APG  only) 

NA 

Comparison  to  ground  truth  data 

0.105-0.615 

Pfa-WAA  surveys 
(manual  pieks) 

<0.25 

Comparison  to  manual  pieks 

0.02-0.90 

Setup  time 

<4  hours 

Setup  time  logged  manually 

0.1-5.45  hours 

Proeessing  time 

<0.25  hr. 

Proeessing  time  logged  manually 

.02-.25 

Detection  location 
accuracy 

90%  <0.5  m 

Comparison  to  ground  truth  data 
and  manual  pieks 

63-95% 

Charaeterization  size 
aeeuraey  (APG  only) 

90%  <0.3  m 

Comparison  to  ground  truth  data 

39-44% 

Charaeterization 
loeation  aeeuraey 
(APG  only) 

90%  <0.3  m 

Comparison  to  ground  truth  data 

90-93% 

Charaeterization  depth 
aeeuraey  (APG  only) 

90  %  <0.3  m 

Comparison  to  ground  truth  data 

94-97% 

4,3  DATA  ASSESSMENT 

The  first  phase  of  the  demonstration  used  the  60-dipole  synthetic  dataset  to  explore  the  parameter 
space  and  optimize  the  algorithms  for  the  four  target  pickers.  We  varied  their  different 
parameters  to  evaluate  and  doeument  their  effeet  and  to  determine  if  certain  “rules  of  thumb” 
eould  be  developed.  This  information  was  used  to  guide  us  when  applying  the  target  piekers  to 
the  datasets  in  the  demonstration.  In  addition  to  this  knowledge,  Phase  1  was  also  used  to  seleet 
a  set  of  starting  parameters  for  eaeh  automatic  picker  that  was  used  in  Phase  2. 

Phase  2  applied  the  target  piekers  to  the  seven  magnetie  datasets  using  the  default  parameters 
output  from  Phase  1 .  Beeause  each  dataset  had  its  own  unique  data  eharaeteristics,  the  starting 
parameters  needed  to  be  adjusted  to  achieve  the  best  performance.  Of  the  automatic  methods, 
the  AS  and  wavelet  method  gave  the  best  results  overall,  but  eaeh  method  had  its  own  strengths 
and  weaknesses  and  the  best  method  to  use  was  very  data-dependent.  A  general  observation  for 
all  the  automatie  pickers  is  that  they  should  not  be  run  blindly.  The  analyst  should  carefully 
ehoose  their  parameters  and  analyze  the  results.  The  process  of  iteratively  ehanging  parameters 
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and  visual  review  of  the  results  was  essential  in  selecting  the  best  parameters  and  producing  the 
best  results. 

The  AS  method  was  the  fastest  and  simplest  method.  It  worked  well  on  a  range  of  target  sizes 
and  densities.  Its  main  weakness  was  picking  targets  in  areas  with  a  variety  of  background 
geology  or  noise  levels  because  a  hard  threshold  is  required. 

The  wavelet  method  worked  well  on  a  range  of  target  sizes  and  densities.  It  performed  better 
than  the  other  methods  at  picking  targets  that  were  clustered.  Its  main  weaknesses  were  the 
complexity  of  the  parameters  and  over  picking  in  areas  with  geology,  noise,  or  poor  data  quality. 

The  clustering  method  performed  well  detecting  isolated  targets  with  similar  sizes.  The 
clustering  algorithm  had  problems  picking  anomalies  in  areas  with  high  geology,  noise,  or  a 
range  of  target  sizes.  There  were  also  multiple  parameters  that  needed  to  be  set. 

The  MF  method  was  hindered  by  its  requirement  to  have  total  data  coverage  within  the  filter 
box.  This  would  increase  geophysical  data  collection  costs  because  additional  data  would  need 
to  be  collected  around  the  perimeter  of  the  survey  area.  It  had  difficulty  with  overlapping 
targets,  range  of  target  sizes,  and  poor  data  quality.  The  one  area  it  excelled  at  was  picking 
targets  in  areas  with  a  variety  of  geology. 

Overall,  the  manual  method  proved  to  be  the  best  at  picking  valid  targets,  especially  in  areas  with 
varying  amounts  of  geology  and  background  noise.  In  general,  the  manual  picker  was  able  to  set 
a  lower  threshold  for  each  dataset  than  the  automatic  methods  because  he  can  screen  out 
anomalies  caused  by  noisy  data  or  geology  and  pick  the  small  amplitude  anomalies  located  in  the 
quieter  areas.  The  main  drawbacks  to  the  manual  method  are  time,  operator  bias,  and  operator 
error.  In  this  demonstration  the  manual  method  was  four  to  50  times  slower  (not  including  setup 
time)  than  the  automatic  methods,  depending  on  the  dataset. 
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5.0  COST  ASSESSMENT 


5.1  COST  REPORTING 

This  demonstration  focuses  on  detecting  anomalies  observed  in  magnetic  data.  As  such,  it 
encompasses  only  a  small  subset  of  costs  that  are  typically  associated  with  full-scale 
demonstrations.  The  relevant  cost  categories  and  actual  costs  for  this  demonstration  are  shown 
in  Table  9. 


Table  9.  Demonstration  Cost  Categories  and  Details. 


Cost  Category 

Details 

Subcategory 

Time 

(hours) 

Data  analysis 

Setup  parameters  for 
identifying  anomalies 

Analytic  signal 

1.5 

150 

Wavelet 

8.2 

820 

Clustering 

23.7 

2,370 

Matched  filter** 

21.9 

2,190 

Manual 

0 

0 

Data  analysis 

Running  automatic  target 
pickers 

Analytic  signal 

1.7 

170 

Wavelet 

5.4 

540 

Clustering 

3.8 

380 

Matched  fdter** 

6.2 

620 

Manual 

43 

4,300 

*Cost  assume  a  fully-loaded  labor  rate  of  $  1 00  per  hour. 

**  Matched  filter  hours  include  only  six  of  the  seven  datasets  because  it  could  not  be  run  on  the  WAA  Pueblo  transect  data. 


This  demonstration  analyzed  seven  data  sets  consisting  of  roughly  7  hectares  of  production 
survey  data  and  82  hectares  of  WAA  data.  Approximately  3,000  anomalies  were  identified  with 
each  picking  method.  The  estimated  costs  required  to  analyze  the  datasets  with  each  of  the 
picking  methods,  excluding  one-time,  demonstration-related  costs  such  as  experimentation, 
optimization,  non-routine  analysis  and  testing,  and  reporting  are  presented  in  Table  10.  The 
costs  associated  with  setting  up  the  parameters  in  Table  10  assume  only  one  setup  and  not  the  7 
setups  required  under  this  demonstration.  The  estimated  cost  was  calculated  by  taking  the 
average  setup  cost.  These  costs  will  be  more  indicative  of  a  real-world  implementation  because 
it  is  likely  only  one  setup  will  be  required.  Also,  the  table  includes  costs  for  using  all  methods  to 
detect  anomalies.  In  the  real  world,  only  one  of  the  automatic  methods  would  be  used  to  select 
anomalies;  therefore,  the  real-world  costs  for  each  method  were  calculated  and  summarized  in 
the  bottom  section  of  the  table. 
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Table  10.  Data  Analysis  Cost  Categories  and  Details. 


Cost  Category 

Details 

Subcategory 

Time 

(hours) 

Data  Analysis 

Setting  up  parameters  for 
different  methods 

Analytic  signal 

.2 

20 

Wavelet 

1.2 

120 

Clustering 

3.4 

340 

Matched  fdter 

3.2 

320 

Manual 

0 

0 

Data  Analysis 

Pieking  targets  using  the 
different  methods 

AnalMic  signal 

1.7 

170 

Wavelet 

5.4 

540 

Clustering 

3.8 

380 

Matched  fdter** 

6.2 

620 

Manual 

43 

4,300 

Subtotal 

Total  for  each  method 

Analytic  signal 

1.9 

190 

Wavelet 

6.6 

660 

Clnstering 

7.2 

720 

Matched  filter** 

9.4 

940 

Mannal 

43 

4,300 

*Costs  assume  a  fully-loaded  labor  rate  of  $  1 00  per  hour. 

**  Matched  filter  hours  include  only  six  of  the  seven  datasets  because  it  could  not  be  run  on  the  WAA  Pueblo  transect  data. 


5.2  COST  ANALYSIS 

The  baseline  alternative  to  the  automatie  target  piekers  is  manual  target  seleetion.  Implementing 
the  automatie  pieking  methods  used  in  this  demonstration  should  eonsiderably  reduee  the  time 
and  thus  eost  required  to  piek  anomalies  when  eompared  to  the  manual  method.  The  amount  of 
eost  savings  will  depend  on  the  data,  whieh  will  determine  the  most  effieient  automatie  method. 
In  areas  with  isolated  anomalies  and  low  baekground  noise  or  geology,  the  eost  savings  will  be 
maximized  beeause  the  automatie  piekers  are  able  to  deteet  over  90%  of  the  anomalies  in  a 
fraetion  of  the  time  eompared  to  the  manual  method.  For  this  best  ease  seenario,  we  will  assume 
a  deteetion  rate  of  95%  using  the  AS  method.  The  setup  time  for  the  AS  method  is  minimal  for 
the  average  dataset,  and  it  runs  the  quiekest  of  the  automatie  methods  evaluated  in  this 
demonstration.  Using  this  demonstration  as  a  guide,  the  AS  method  would  eost  approximately 
$200  eompared  to  $4,300  for  the  manual  method.  If  the  manual  method  is  used  to  seleet  the  final 
5%  of  the  targets  and  we  assume  the  additional  eosts  will  be  equivalent  to  15%  of  the  total  eosts 
for  the  manual  method,  the  eost  savings  will  be  about  80%  eompared  to  using  only  the  manual 
method. 

As  the  geologie  noise  inereases  or  data  quality  deereases,  the  eost  savings  will  dimmish  but  still 
should  be  signifieant.  Even  the  toughest  datasets  in  this  demonstration  will  produee  eost  savings. 
For  example,  the  airborne  WAA  dataset  at  Pueblo  eontained  an  abundanee  of  geology  that 
resulted  in  Pds  of  around  0.5  for  the  automatie  piekers.  The  AS  required  less  than  an  hour  to 
setup  and  run  while  the  manual  method  took  9  hours.  Even  if  it  takes  an  additional  5  hours  of 
manual  pieking  to  seleet  the  other  half  of  the  targets,  the  time  savings  will  be  3  hours  or  33%. 
The  eost  savings  may  appear  to  be  signifieantly  redueed  if  different  methods  sueh  as  the  ME  or 
elustering  algorithm  are  used  beeause  of  the  inereased  time  to  setup  the  final  parameters  or  the 
longer  proeessing  time.  But  the  proeessing  time  is  mainly  a  funetion  of  the  eomputer  run  time, 
and  the  data  analyst  is  able  to  do  other  tasks  while  the  algorithms  are  running.  Earge  datasets 
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that  will  take  a  long  time  to  eomplete  ean  be  run  in  bateh  mode  at  off  hours,  whereas  the  manual 
method  is  entirely  hands-on  and  labor-intensive. 

The  parameter  setup  time  represented  a  large  portion  of  the  proeessing  time  for  the  relatively 
small  datasets  used  in  this  demonstration.  The  ground-based  surveys  ranged  from  0.3  to  10  aeres 
in  size.  On  the  other  hand,  large-seale  eleanup  efforts  are  typieally  hundreds  of  aeres  so  the 
setup  eosts  will  be  a  trivial  portion  of  the  total  eosts.  Assuming  the  geology,  data  quality,  and 
data  objeetives  for  the  survey  area  are  relatively  eonstant,  the  automatie  target  pieking 
parameters  should  need  minimal  ehanges.  If  we  eompare  only  the  time  to  aetually  run  the 
automatie  methods  to  the  manual  method,  we  find  the  proeessing  time  of  the  automatie  piekers  is 
roughly  10%  of  the  time  for  the  manual  method.  The  aetual  eost  savings  is  greater  assuming  the 
data  analyst  is  attending  to  other  tasks  while  the  automatie  methods  are  proeessing.  If  we  faetor 
in  10-40%  additional  time  to  fill  in  missing  anomalies  using  the  manual  method,  there  is  still  a 
eost  savings  of  50-80%  aehieved  by  using  the  automatie  pieking  methods. 
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6.0  IMPLEMENTATION  ISSUES 

6.1  COST  OBSERVATIONS 

The  target  pieking  methods  demonstrated  here  are  dependent  on  data  quality,  target  size, 
geology,  SNR,  and  target  density.  Magnetie  data  that  possess  a  high  SNR  and  aeeurate  spatial 
positions  will  produee  higher  Pds  and  potentially  fewer  false  alarms.  Although  high  quality  data 
is  desired,  some  of  the  automatie  pieking  methods  performed  well  in  this  demonstration  using 
data  with  fiducial-based  positions. 

The  site-specific  factors  that  will  have  a  large  impact  on  the  potential  cost  savings  are  the  range 
of  target  sizes,  target  density,  and  geology.  As  mentioned  in  this  report,  many  of  the  automatic 
target  picking  algorithms  required  a  threshold  parameter  to  be  set.  This  causes  a  major  problem 
in  areas  with  varying  amounts  of  geology  and  background  noise.  Setting  the  threshold  at  a  level 
to  detect  the  smallest  targets  produces  many  false  positives  due  to  geology,  whereas  many  targets 
will  be  missed  if  the  threshold  is  set  above  the  background  geology.  An  increase  in  anomalies 
that  need  manual  intervention  will  decrease  the  potential  cost  savings  realized  by  using  the 
automatic  methods. 

The  analysis  costs  may  be  marginally  reduced  as  the  learning  curve  on  the  picking  methods 
increases.  The  experienced  data  analyst  will  be  able  to  determine  more  quickly  which  method  to 
use  for  a  given  dataset.  The  experienced  analyst  will  also  be  able  to  set  up  the  optimum 
parameters  in  less  time  and  be  able  to  ascertain  if  they  need  to  be  altered  due  to  changing  site 
conditions.  For  large-scale  surveys,  these  setup  costs  are  minimal  in  comparison  to  the  entire 
target  picking  process.  The  picking  method  that  is  ideal  for  a  site  should  not  significantly  alter 
the  total  cost  because  the  difference  in  setup  time  is  a  one-time  expense,  and  the  additional 
processing  time  for  different  methods  affects  only  the  computer  run  time  and  not  analysts’  labor. 

6.2  PERFORMANCE  OBSERVATIONS 

The  primary  performance  criteria  in  this  demonstration  are  Pd,  Pfa,  location  accuracy,  and  target 
picking  time.  In  other  words,  the  automatic  methods  should  detect  a  high  percentage  of  targets 
with  minimal  false  alarms  in  a  small  amount  of  time  compared  to  the  manual  method.  Many  of 
the  factors  (target  density,  target  size,  geology)  that  determine  the  ultimate  success  of  the 
automatic  pickers  are  not  under  our  direct  control.  The  main  reason  that  some  automatic  pickers 
did  not  reach  the  performance  goals  was  related  to  signal-to-noise  problems.  In  most  cases,  there 
was  too  much  noise  caused  by  geology  in  the  area  to  achieve  a  high  Pd  without  a  high  Pfa  so  the 
Pd  was  sacrificed  to  limit  the  Pfa.  Even  in  these  situations,  cost  savings  will  be  achieved 
compared  to  using  only  the  manual  method. 

6.3  SCALE-UP 

There  are  no  critical  issues  with  regard  to  scaling  up  the  demonstration  costs  reported  here  to 
larger,  full-scale  implementations.  Although  the  algorithms  are  able  to  process  large  amounts  of 
data,  their  computational  efficiency  is  dependent  on  the  amount  of  RAM  available.  This  should 
not  be  a  problem  because  most  target  picking  on  large-scale  surveys  are  typically  done  on  a 
smaller  grid-by-grid  basis  that  is  easily  supported  by  a  computer  with  standard  specifications. 
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6.4  OTHER  SIGNIFICANT  OBSERVATIONS 


There  are  no  eritieal  issues  with  regard  to  the  implementation  of  the  automatie  target  pieking 
methods.  All  methods  are  eurrently  operational  under  the  Oasis  montaj  software  platform  that  is 
used  by  a  large  pereentage  of  the  UXO  eommunity.  The  target  pieking  algorithms  would  ideally 
be  transitioned  to  the  user  eommunity  through  Geosoft  as  part  of  their  proposed  UXO  System. 

6.5  LESSONS  LEARNED 

Of  the  automatie  methods,  the  AS  and  wavelet  method  gave  the  best  results  overall,  but  eaeh  of 
the  methods  had  their  own  strengths  and  weaknesses,  and  the  best  method  to  use  was  very  data- 
dependent.  A  general  observation  for  all  the  automatie  piekers  is  that  they  should  not  be  run 
blindly.  The  analyst  should  ehoose  parameters  earefully  and  analyze  the  results.  The  proeess  of 
iteratively  ehanging  parameters  and  visual  review  of  the  results  was  essential  in  seleeting  the  best 
parameters. 

The  best  use  of  the  automatie  target  piekers  may  be  to  quiekly  and  eonsistently  seleet  all  the 
strong  isolated  targets  by  setting  the  pieking  parameters  to  minimize  false  alarms.  The 
remaining  anomalies  ean  then  be  seleeted  using  the  manual  method.  This  eombines  the  strengths 
of  eaeh  method  and  will  result  in  a  better  produet  eompared  to  using  only  the  automatie  methods 
and  will  save  eosts  eompared  to  using  only  the  manual  method. 

6.6  END  USER  ISSUES 

The  end  users  of  this  data  analysis  teehnology  inelude  private  eontraetors  who  eonduet 
geophysieal  investigations  in  support  of  UXO  eleanup  programs  and  government  employees  who 
provide  teehnieal  oversight.  This  demonstration  will  introduee  the  stakeholders  and  end  users  to 
the  applieability  of  different  automatie  target  piekers  to  their  data.  This  basie  information  will 
help  improve  the  results  and  eost  savings  of  future  geophysieal  investigations  eondueted  by 
others. 


6.7  APPROACH  TO  REGULATORY  COMPLIANCE  AND  ACCEPTANCE 

At  a  given  site,  the  methods  and  teehniques  for  a  proposed  geophysieal  survey  to  deteet  buried 
munitions  needs  to  be  tested  and  evaluated.  As  part  of  this  proeess,  a  geophysieal  prove-out  is 
eonstrueted  and  used  to  identify  the  eapabilities  of  the  proposed  teehnologies.  At  this  time,  the 
results  and  parameters  of  the  different  automatie  pieking  methods  ean  be  evaluated.  A  dig  sheet 
identifying  potential  targets  in  the  geophysieal  prove-out  as  well  as  a  deseription  of  the 
teehnology  and  optimum  parameters  seleeted  is  submitted  to  the  Corps  of  Engineers  for  review 
and  approval. 
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